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ABSTRACT 

This paper discusses a formative evaluat ion. method by 
which to assess World-Wide-Web (WWW) documents* Also presented are 
the results from a pilot evaluation which examined users’ 
interactions with Web documents and the effects of providing 
"document maps" to guide users to specific information* A video-split 
screen technique was used to record users interacting with the WWW 
and to assess the design of associated documents* Three female and 
two male subjects participated in the study; four were undergraduate 
students and one was a graduate student. Subjects were given two 
tasks: (1) to browse through the WWW and locate specific 
inf ormation/documents ; and (2) to use and assess Eastern Illinois 
University’s Department of Media Services Web site. At the completion 
of the session, subjects filled out a survey which collected 
information on their reactions toward the usefulness of the "document 
maps" and data related to their computer experience and knowledge of 
the WWW* Results indicated that the video-split screen method can be 
an effective means to assess WWW documents. Subjects need to be shown 
that a "document map” exists for their use and they must be 
explicitly informed as to its purpose and function. As WWW is used 
for the delivery of instruction, it is important to provide learners 
with a means of instruct ional support* Three figures present sample 
transcripts from the evaluation sessions and depict the time spent 
searching (by trials) and the number of times maps were used (by 
trial)* (Contains 18 references.) (AEF) 
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Introduction 

Hypermedia programs are 
computer-based software systems for 
organizing and storing information. They 
are structured as an arrangement of nodes 
(concepts in text or graphical form) 'and 
links (semantic relationships between 
concepts), and allow users to obtain 
mediated experiences non-sequentially 
based on individual need and interest 
(Kumar, Helgeson & White. 1994). 
Users can create unique lesson structures 
reflective of individual learning 
requirements (Park & Hannafin, 1993). 
Early computer-based learning programs 
often forced users to page sequentially 
through information one screen at a time 
from beginning to end. Today's hypertext 
and hypermedia programs, on the other 
hand, typically provide clickable objects 
(e.g.. text, pictures, etc.) as a means to 
access additional and./or related 
information. For educational purposes, 
these environments have the potential to 
represent knowledge (Bower & Hilgard. 
1 98 1). Accordingly, they may help to 
facilitate information encoding and 
retrieval (Park & Hannafin, 1 993) and 
more appropriately serve human 
information processing needs. 

Proponents assert that hypermedia 
programs are ideal for adaptive instruction. 
They contend that students can discern 



which information to study and thus can 
organize it to match their cognitive learning 
capabilitieis (Park, 1 99 1). While hyper- 
media programs provide greater learner 
control, a well-cited problem with these 
environments is that users frequently "get 
lost" or disoriented (see Conklin. 1987; 
Neilson. 1 989; Edwards and Hardman 
1 989). As the number of nodes and the 
complexity of their linkages in the 
hypermedia program increase, it is not 
unusual for the users of a complex 
hypermedia program to get lost in the 
middle of the program (Park. 1 99 1). As 
Stanton and Baber (1 994) point out. the 
phenomenon of getting "lost" can take 
several forms: users not being able to find 
required information or becoming 
disoriented, losing their way or being 
unable to formulate appropriate actions. 
This problem is fundamental to the design 
of hypermedia programs. It may be 
compounded in networked environments 
such as the World-Wide-Wcb (WWWi 
which have the capability to expand access 
world-wide presenting users with a 
seemingly infinite number of nodes and 
links. 

While world-wide hypermedia 
systems are having tremendous impact on 
information access and retrieval, their 
development, it .seems, is often influenced 
more by the technological capabilities of 
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computers rathei than by visual and 
interface design principles. Such 
environments are fnjquently designed as 
scrambled, content-free information or 
gibberish and more complex systems 
being developed appear to compound this 
problem (Stanton & Baber, 1994). 
Technologically, developers have the 
capability to include pictures, text, audio 
and video in documents and to almost 
instantaneously link to sites throughout the 
world. However, design protocols often 
differ from one site to the next which can 
confuse users and increase the possibility 
of them getting disoriented or "lost". For 
example, browsers of the World-Wide- 
Web are likely to find many dissimilar 
home page designs. While "web 
browsers" (e.g., Netscape, Mosaic) 
provide navigational options, home page 
developers often add navigational elements 
which confuse and mislead users and are 
not consistent from one site to the next. 
Developers should attempt to understand 
.how users browse information and how 
the design of documents influences user 
navigation. 

Purpose of the Paper 

Complex interactive hypermedia 
pose significant challenges for those 
researching its effect on users (Gay & 
Mazur, 1993). For example, it is no 
simple task to design a world-wide access 
hypermedia system which all users find 
intuitive and easy to use. Understanding 
how users interact with these systems is 
important because their success is often 
contingent on an appropriate human- 
computer interface. However, while 
powerful from an information processing 
perspective, flexible, non-sequential, user- 
centered designs make it more complex to 
examine how users interact with a system 
(Gay & Mazur, 1993). Moreover, as the 
utilization of media objects such as video, 
audio, and animation increases, so does 
the difficulty of designing an effective user 
interface (Wadlow, 1990, p. 181). 

To better understand the human- 
computer interface, recent formative 
evaluation approaches have taken a more 



holistic view (Winograd & Flores, 1986). 
Many approaches advocate qualitative 
methodologies (Card, Moran & Newell, 
1983) using multiple data collection 
instruments (Marchionini, 1990) in an 
attempt to develop an understanding of 
how computer-based systems are used. 
The video-split screen (Gibbs & Shapiro, 
1994) is one such technique for 
monitoring the interactions between user 
and computer. This method video records 
subjects' actions and decisions in a 
program, their verbal commentary and 
elaboration, their observations, and 
nonverbal and attitudinal reactions. 
Through a video effects generator, two 
images (subject and computer screen) are 
integrated so that the subject's behavior 
can be observed simultaneously with the 
computer screen. Subject and computer 
screen are each simultaneously video- 
recorded. This technique provides a 
permanent visual and auditory account of 
evaluation for subsequent analysis. 
Among other things, this data helps to 
identify users' reactions, needs, use 
patterns, and interactions and portrays, to 
some extent, a more fuller representation 
of them in the learning environment. 
From it, researchers can potentially design 
more powerful environments. 

In recent years, there has been 
much interest in methods that help 
researchers understand cognition and the 
interplay between learners and computer- 
based learning environments. With 
techniques such as the video-split screen, a 
learner's interactions with a computer can 
be collected using multiple data collection 
instruments and analyzed to obtain a fuller 
awareness of their thought processes. 

This paper discusses a formative 
evaluation method by which to assess 
World-Wide-Web documents. Also 
presented are results from a pilot 
evaluation which examined users' 
interactions with Web documents and the 
effects of providing "document maps" to 
guide users to specific information. 
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Methods 

A video-split screen technique was 
used to record users interacting with the 
World-Wide-Web and to assess the design 
of associated documents. Three female and 
two male subjects participated in the 
evaluation sessions. Four of the 
participants were undergraduate students 
and one was a graduate student. Most of 
them had used the World-Wide-Web no 
more than three to four times prior to the 
evaluation sessions. 

Observation Technique 

Subjects were given two tasks; 1) 
to browse the World-Wide-Web and locate 
specific information/documents and 2) use 
and assess Eastern Illinois University's 
Department of Media Services Web site. 
On each task, subjects were asked to think 
aloud. They took as much or as little time 
as they needed. Periodically, the 
researchers prompted them to speak up or 
tel! what they were thinking. 

Task 1 

On this task, subjects had three 
trials. For each trial they had to find one 
document located within a web site. 
Printouts of the documents to be found 
were given to the subject. The site chosen 
for this task was the White House; 
however, many other sites could have 
been substituted for it. 

On trials 1 and 3, "document 
maps" were provided to assist with 
information searches. These maps 
consisted of a hierarchical breakdown (in 
text form only) of topics that directed 
subjects to the document for which they 
were searching. The map was positioned 
in a window behind the Web page and 
would display when clicked with the 
computer's mouse. No map was available 
for the second trial. 

Task 2 

For this task, subjects used Eastern 
Illinois University's Department of Media 
Services' home page and talked aloud 
while interacting with it. Subjects were 
asked to critique the home page, suggest 



revisions, discuss misunderstandings or 
provide any commentary that would 
contribute to improving the document. 
They could spend as much time or as little 
time as they wanted on this task. 

At the completion of the session, 
subjects filled out a survey which collected 
information on their reactions toward the 
usefulness of the "document maps". The 
survey also collected data related to 
subjects' computer experience and 
knowledge of the World-Wide-Web. 

Results 

This was a pilot study with a small 
number of subjects. Given this constraint, 
no attempt is made to generalize the 
findings discussed below to other groups. 
Nevertheless, the study yielded 
informative data worthy of additional 
research. 

It appears that the video-split 
screen method can be an effective means to 
assess WWW documents. The video 
recordings allowed the researchers to 
visually and aurally reconstruct the actions 
taken by subjects. This is beneficial for 
several reasons. First, it enables one to 
simultaneously monitor, among other 
things, how users navigate in a system, 
the types of options selected and the order 
in which they are selected. It also shows 
where users appear confused or where 
they get lost, what they like and dislike, 
and their observations about the program. 
Second, the technique allows for learners' 
immediate feedback about their interactions 
with the program. Other forms of 
assessment may not fully reflect a 
subject's instantaneous processing of 
stimuli or the way in which one explores 
and links conceptual events presented by 
the hypermedia system. Third, assessing 
a system can be a laborious task when 
users are asked to record/writc their 
comments and reactions. This method of 
assessment removes the burden of 
evaluation from subjects and allows them 
to more easily provide feedback. Lasth. 
the technique permits the review and 
validation of observations by several 




‘i’ 



261 



reviewers. Sample transcripts, resulting 
from the evaluation sessions, are presented 
in Figure 1. Transcripts made from the 
video recordings provide a useful resource 
for analyzing subject-computer interaction. 

Figure 1 

Structured Observation 
Transcript Sample 



■’All this extra space and all these little pictures. 
They don't really draw your eye ..." 

"I was just kind of wondering where I am. Uh, it 
looks like I started something major just by all 
the data going back and forth." 

"I like the purple and blue, the colors are really 
bright, they catch your eye, it looks neat. It 
seems like it gets kind of repetitive, everywhere 
you go you end up just seeing the staffs names, 
the name of the staff personnel." 

I don't know what else is supposed to be on here, 
the interactive learning, I thought that would be a 
little game or something to play - to learn how to 
use the computer, but it wasn't." 



than on the second (no map available). 
Three subjects had 4 or more years of 
computer experience. Despite this 
computer familiarity, the search tasks, 
especially trial 2, posed difficulty for 
them. It should be pointed out, however, 
that experience with computers does not 
necessarily mean that one will be able to 
effectively search the Web having little 
prior exposure to it. 

Figure 2 

Time Spent Searching 
(By Trials) 



Trials 




Minutes in Search 




Total Time 
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On three occasions, subjects were 
asked to find one document located within 
the White House WWW site. They were 
given printouts of the three documents and 
asked to locate them on the web. On the 
first and third trials a "document map" was 
provided. No map was available for the 
second trial. 

There appeared to be differences in 
the way subjects reacted during search 
trials depending on the availability of the 
"documents map". In the second trial, for 
example, there was no map and most of 
the subjects showed what appeared to be 
impatience soon after their first few 
attempts to find the document failed. Three 
of the subjects showed uneasiness after 
two minutes. They looked for the 
information aimlessly, proceeding forward 
and backward through many screens and 
quitting the task before finding the 
document. Figure 2 indicates that most 
subjects took less time to find documents 
on the first and third trials (maps available) 



Subjects stated that the "document 
map" was very useful when searching for 
documents. From observations, it 
appeared that subjecis attempted to first 
find the documents without the aid of the 
map. However, once it was clear that the 
map provided a visual guide to the to-be- 
found documents, their use increased. 
Three subjects use of the "document 
maps" increased from the first to the third 
trial. Use was determined by counting the 
number of times subjects viewed the map 
on each trial. On trial 1, subjects 3, 4 and 
5 used the maps a total of three times. 
Their use of the map increased to a total of 
16 times on ihe third trial (see Figure 3). 
Subjects 1 and 2 used the map less on the 
third trial than on the first. 



262 



r 

o 



Figure 3 

Number of Times Maps Were Used 
(By Trial) 
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During trials, subjects initially 
seemed to search for context and 
informational clues rather than visual clues 
and some resorted to aimlessly clicking 
navigation options only after all other 
approaches failed. This has implications 
for designers in that the informational 
content presented should be supported 
and/or coincide with visual cues and 
navigational elements. For example, while 
certain visual cues attracted attention, 
subjects also read the text on-screen and 
from this made inferences as to where they 
might find the documents for which they 
were searching. Thus, it was not a 
haphazard random process by which 
subjects searched for information — the 
meaning of the presented information was 
seiiously considered and directed their 
sea.'ch efforts. 

Discussion 

As the World-Wide-Web is used 
for the delivery of instruction, it is 
important to provide learners with a means 
of instructional support. This is 
particularly important, given that users 
frequently experience disorientation within 
hypermedia systems. For instructional 
purposes, learners need information about 
how to find topics and concepts under 
study. Creating "document maps" for use 
with the World-Wide-Web can be a time 
consuming process considering the 
seemingly infinite number of nodes and 
links that could be established. However, 
from this pilot study its seems even the 
simplest of maps are useful. When a finite 



path is established to reach a particular 
topic, a simple hierarchical breakdown of 
the topics will be of much value to 
learners. It provides them with a guide and 
appears to prevent them from getting side- 
tracked by distracting information. 

From this study, it was observed 
that subjects need to be shown that a 
"document map" exists for their use. They 
also must be explicitly informed as to its 
purpose and function. In some cases, it 
was not until several failed attempts and 
much frustration that subjects realized that 
they had the map. Therefore, for 
instructional purposes, such resources 
should be made identifiable and easily 
accessible to learners. It would have 
helped if subjects had a "Read Me First" 
button which informed them about the 
purpose of the map. 

In this study, a map was provided 
by a separate application and was not 
clickable. Nevertheless, subjects made 
repeated attempts to click the words on the 
map in attempts to go to a destination. 
When no action resulted from their clicks, 
some appeared confused and experienced 
difficulty getting back to the search. 
Therefore, whenever possible, designers 
should provide navigational elements or 
clickable objects on maps so that when 
users click them, they are taken to the 
desired location. 

Summary 

This paper has discussed an 
approach for assessing World-Wide-Web 
hypermedia systems. Recent assessment 
approaches have become multifaceted and 
holistic in an attempt to more fully 
understand ,.he human-computer interface. 
Data collection instruments, such as the 
one discussed in this paper, have the 
potential to provide more direct 
comprehensive evaluations of complex 
systems. 

A current problem with hyper- 
media programs is that users frequently 
"get lost" when using them, Developers 
must consider methods to guide users to 
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information and to reduce the potential of 
them becoming disoriented. From the 
analysis presented, informational maps 
will likely increase the efficiency with 
which learners search for information. 
When designed effectively, they appear to 
have positive impact on students. 
Moreover, these maps can present the 
structure of to-be-learned information and 
thus have positive learning benefits. 

Further Research 

From this pilot, a more in-depth 
study is planned to examine the effects of 
maps on users' browsing behavior and 
cognitive processing. Using the techniques 
described in this paper, it may be possible 
to determine the effects of various types of 
maps and WWW interface designs on 
users' cognitive processing. Our 
knowledge of learners' thought processes 
in various types of computer-based 
learning environments is limited (Nakhleh 
& Krajcik. 1991). Using techniques like 
the video-split screen with methods to 
analyze subjects' verbal commentary such 
as protocol analysis (Ericsson & Simon. 
1984) may prove useful in better 
understanding learners' cognitive 
processes while engaged in a hypermedia 
program. From this knowledge, more 
effective design may result. 
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